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Agenda 


■  How  do  I  know  if  my  model  is  of  good  quality? 

■  What  is  quality? 

■  Model-Based  Engineering 

-  SysML  and  UML 

■  Examples: 

-  Requirements  traceability 

-  Style,  Standards  and  Visualization 

-  Parametrics 

-  Measures  of  Effectiveness 

-  Trade-off  Analysis 

-  Model  Execution 

■  Quality  and  Process 

■  Questions? 
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How  do  I  know  if  my  model  is  of  good  quality? 


■  A  valid  question  whether  for : 

-  Yourdon,  IDEF,  UML,  SysML,  DoDAF,  NAF,  and  Lego  models. 

■  Can  be  both  subjective  and  objective 

■  Assessment  criteria  is  essential  for  project  success 

-  Without  assessment  criteria  the  model  will  become  unfocussed 

-  The  most  important  criterion  is  whether  or  not : 

it  communicates  its  intent 

-  Goes  to  the  heart  of  why  the  model  was  created  in  the  first  place 


C  a 
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What  was  the  question? 


■  “AH  models  are  wrong,  some  models  are  useful.” 

Professor  P.E.  Box 

■  Models  are  an  abstraction  of  the  problem  or  solution  space 

-  Reflect  an  abstraction  of  one  or  more  viewpoints 

■  A  model  should  be  created  to  answer  one  or  more  questions 

-  Performance 

-  Functionality 

-  Timing 

-  Structure 

-  Usability 

-  Project  lifecycle 

-  Product  lifecycle 

-  Etc. 
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What  is  quality? 


■  Dictionary  definitions  are  neutral 

-  1 .  A  distinguishing  characteristic  or  attribute. 

-  2.  The  basic  character  or  nature  of  something. 

-  3.  A  feature  of  personality. 

-  4.  Degree  or  standard  of  excellence. 

-  5.  High  social  status. 

-  6.  Musical  tone  color. 

-  7.  Excellent  or  superior;  a  quality  product. 


■  Thesaurus  synonyms  are  largely  positive 

-  Character 

-  Sort 

-  Tendency 

-  Excellent 

-  Goodness 
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Quality  in  Systems  Development 


■  “A  scalar  attribute  reflecting  ‘how  well’  a  system  functions.” 

“Examples  include  Availability,  Usability,  Integrity,  Adaptability,  and 
many  others.” 

-  Quality  levels  are  capable  of  being  specified  quantitatively  (as  are  all 
scalar  attributes) 

-  Quality  levels  can  be  measured  in  practice 

-  Quality  levels  can  be  traded  off  to  some  degree;  with  other  system 
attributes  valued  more  by  stakeholders 

-  When  quality  levels  increase  towards  perfection,  the  resources 
needed  to  support  those  levels  tend  towards  infinity 

Planguage  concept  glossary,  Tom  Gilb 

■  The  desired  level  of  quality  varies  according  to  the  model  purpose 

-  Project  bid 

-  Safety  critical  implementation 

-  Brainstorming  session 

-  Etc. 
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Model-Based  Engineering 


INCOSE 

International  Council  on  Systems  Engineering 


■  Model-based  Systems  Engineering  (MBSE)  is  the  formalized 
application  of  modeling  to  support  system  requirements,  design, 
analysis,  verification,  and  validation  activities  beginning  in  the 
conceptual  design  phase  and  continuing  through-out  development  and 
later  lifecycle  phases.”  (INCOSE,  2007). 

■  Modeling  is  at  the  heart  of  all  aspects  of  the  development  effort 

-  Covers  the  complete  product  and  project  lifecycle 

-  Has  a  direct  effect  on  any  generated  artifacts. 

-  MBE  encompasses  architecture,  systems  and  software 
development. 
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SysML  and  UML 


UNIFIED 

MODELING 

LANGUAGE 


■  The  Unified  Modeling  Language  (UML). 

-  “  UML  provides  system  architects  working  on  object  analysis  and 
design  with  one  consistent  language  for  specifying,  visualizing, 
constructing,  and  documenting  the  artifacts  of  software  systems,  as 
well  as  for  business  modeling”.  (OMG,  1999) 

-  Now  the  de  facto  standard  for  modelling  software  systems 

-  UML  consists  of  class,  use  case,  component,  deployment,  state 
machine,  sequence,  timing,  activity,  package,  communication, 
composite  structure,  interaction  overview  and  instance  diagrams. 
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SysML  and  UML 


OMG 

SYSTEMS 

MODELING 

LANGUAGE 


■  The  Systems  Modeling  Language  (SysML) 

-  “Supports  modeling  of  a  broad  range  of  systems  which  may  include 
hardware,  software,  data,  personnel,  procedures  and  facilities.  “ 

-  “Used  to  analyze,  specify,  design  and  verify  complex  systems, 
intended  to  enhance  systems  quality,  improve  the  ability  to  exchange 
systems  engineering  information  amongst  tools  and  help  bridge  the 
semantic  gap  between  systems,  software  and  other  engineering 
disciplines.”  (OMG  SysML,  2003). 

-  SysML  added  two  new  diagrams 

-  parametric  and  requirements  diagrams. 

-  Class  and  composite  structure  diagrams  were  modified 

-  Include  elements  to  model  logical  and  physical  block  structures  for 
systems  engineers. 

-  In  SysML  they  are  called  respectively: 

-  Block  Definition  Diagram  (BDD)  and  Internal  Block  Diagram  (IBD) 
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Changes  in  Systems  Engineering  Practice 


Change  from  Document  centric  to  Model  centric 


Requirement  Specifications 
Interface  Definitions 
System  Architecture 
System  Functionality 
Trade-off  Analysis 
Test  Specifications 


Old  Approach 


New  Approach 
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The  Four  Pillars  of  SysML  (ABS  Example) 


1.  Structure 

bdd  [Package]  Vehicle  [ABS] 


ibd  [Block]  Anti-Lock  Controllerl 


«Block» 

«Block» 

«Block» 

Library:: 

Anti-Lock 

Library:: 

Electronic 

Controller 

Electro-Hydraulic 

Processor 

Valve 

dl 


ml 


«Block» 

«Block» 

Traction 

Brake 

Detector 

Modulator 

«Block» 

Anti-Lock  Controller 


2.  Behavior 


nr 


ado  r  a  ..-4. ; o  ^  il 


definition 


par  [constraint]  StraightLineVehicleDynamics  [Parametric  Diagram] 


{f=jtf*bf)*d-ti» 


{F  =  ma} 


:  BrakingForceEquation 

P  d 

□ 

g  . 


:  AccelerationEquation 


□ 


□ 


□ 


4.  Parametrics 


:  DistanceEquation 

n  r 


:  VelocityEquation 


l 


r 

a 


{v  =  dx/dt} 


{a  -  dv/dt} 


iter  face 


S 


«BlockProperty» 
dl  :  Traction  Detector 


«BlockProperty» 
ml  :  Brake  Modulator 


use 


act  PreventLockup 


in 


V 


f 

- 

Detect  Loss  Of 

Traction 

v _ 

) 

-> 


TractionLoss 


-> 


V  activity/ 
funfctiom 


Modulate 
Braking  Force 


v 


req  [Package]  Vehicle  Specifications  [Braking] 


Vehicle  System 
Specification 


«requirement» 
Stopping  Distance 

id# 


102 


txt 


The  vehicle  shall  stop  from 
60  mph  within  150ft  on  a 
clean  dry  surface. 


¥ 


3.  Requirements 


Braking  Subsystem 
Specification 


«requirement» 
Anti-Lock  Performance 


id# 


337 


txt 

The  Braking  subsystem  shall 
prevent  wheel  lockup  under 
all  braking  conditions. 


«deriveReqt» 
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Cross  Connecting  Model  Elements 
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Examples  for  estimating  quality 


■  Requirements  traceability 

■  Style,  Standards  and  Visualization 

■  Parametrics 

■  Measures  of  Effectiveness  /  T rade-off  Analysis 

■  Model  Execution 
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Requirements  Traceability 


■  The  degree  to  which  the  inherent  characteristics  of  process,  product  or 
system  meet  "The  Requirements"  is  the  quality  of  process,  product  or 
system,  irrespective  of  the  sub-classification  or  sub-categorization  of 
"The  Requirements".  ISO  9000 

■  Quality  is,  therefore,  a  question  of  degree. 

-  How  well  does  this  set  of  inherent  characteristics  comply  with  this  set 
of  requirements? 

-  The  quality  of  something  depends  on  a  set  of  inherent  characteristics 
and  a  set  of  requirements  and  how  well  the  former  complies  with  the 
latter. 

-  The  quality  of  something  cannot  be  established  in  a  vacuum. 

-  For  ISO  9000  quality  is  always  relative  to  a  set  of  requirements. 
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Requirements  Traceability 


■  CMMI  Level  2  also  defines  Requirements  Management: 

■  Requirements  Management  (REQM) 

-  Purpose 

-  The  purpose  of  Requirements  Management  (REQM)  is  to  manage  the 
requirements  of  the  project's  products  and  product  components  and  to 
identify  inconsistencies  between  those  requirements  and  the  project's 
plans  and  work  products. 

-  Specific  Practices  by  Goal 

-  SG  1  Manage  Requirements 

-  SP  1.1  Obtain  an  Understanding  of  Requirements 

-  SP  1 .2  Obtain  Commitment  to  Requirements 

-  SP  1.3  Manage  Requirements  Changes 

-  SP  1.4  Maintain  Bidirectional  Traceability  of  Requirements 

-  SP  1.5  Identify  Inconsistencies  Between  Project  Work  and  Requirements 
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The  SysML  Requirements  Diagram 


■  Captures  requirements  hierarchies  and  the  derivation,  satisfaction, 
verification,  copy,  trace,  and  refinement  relationships. 

-  Relate  requirements  to 

-  one  another 

-  system  design  model  elements 

-  test  cases. 

-  The  «rationale»  concept  used  to  annotate  any  model  element  to 
identify  supporting  rationale  including  : 

-  analysis  and  trade  studies 

-  derived  requirement 

-  Design  decision,  etc. 

■  The  requirement  diagram  provides  a  bridge  between  typical 
requirements  management  tools  and  the  system  models. 

■  Reports  and  analysis  can  be  generated  to  show  traceability 
completeness,  traceability  trees,  etc. 


©  2010  Atego.  All  rights  reserved. 


16 


The  SysML  Requirements  Diagram 


req  [Package]  Cruise  Control  System  [Fragment] 


« requirement* 

REQ_CCS_05 

«requirement» 

REQ_CCS_06a 

txt 

Once  the  CCS  is  engaged,  to  activate  cruise  control  the  driver 
can  'set'  the  desired  speed.  Once  this  is  set  the  CCS  shall  take 
over  control  of  the  throttle. 

txt 

When  cruise  control  is  engaged,  the  driver  must  be  able  to 
increment  the  desired  speed  in  increments  of  1  MPH. 

.  «deriveReqt» 

i 


«requirement» 
REQ  CCS  01 


txt 


The  CCS  must  allow  a  driver  to  enable  the  Vehicle  to  maintain  a 
desired  speed. 

A  A 


■*refrie» 


«verify» 


.  Maintain  Speed  . 


«satisfy» 


«  Rationale 


% 


«requirement» 
REQ  CCS  06 


txt 


Maintain  Speed 


When  cruise  control  is  engaged,  the  driver  must  be 

able  to  increment  or  decrement  the  desired  speed 
- - 

i  « satisfy* 


^satisfy* 


i 


\ 

~  % 


«testCase» 

«block»  1 

[Package]  Maintain  Speed  -  with  flows 

Cruise  Control  System 

n 


©  2010  Atego.  All  rights  reserved. 


17 


Standards,  Style  and  Visualization 


■  As  there  are  coding  and  documentation  standards,  there  needs  to  be 


modeling  standards. 

-  Completeness  and  global  checks 

-  SysML  models 

-  Quality  checks  for  model  for  code  generation 
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Standards,  Style  and  Visualization 


■  As  there  are  coding  and  documentation  standards,  there  needs  to  be 


modeling  standards. 

-  Simple  completeness  checks 

-  All  required  fields  have  been 
filled  in 

-  Use  cases  should  contain  a 
full  use  case  description,  pre 
and  post  conditions,  intent  and 
alternate  courses. 

-  More  complex  tests 

-  Use  cases  have  been 
elaborated  to  sequence 
diagrams 

-  Trace  to  or  refine  functional 
requirements 

-  Use  case  text  follows  the 
company  standard  PDL. 


r*  New  UPDM  model,  Version  0  -  Artisan  Reviewer  ] 


File  Edit  View  Reviews  Help 
QBack  Q  f,,y.  #  Print 


I;i  Run  Report  Complete 


Dictionary 


Abstract  class 

Abstract  classes  are  not  supported  by  our  development  language  and  will  not  be  used. 

Abstract  class  not  used  as  a  data  type 

Abstract  classes  should  be  used  as  parameters,  attributes  and  return  types  or  to  encapsulate  common  functionality. 

Abstract  class  without  subclass 

Unless  a  framework  is  being  developed  abstract  classes  should  have  subclasses  so  they  can  be  used. 

Abstract  operation  defined 

Abstract  operations  should  not  have  their  functionality  defined  on  sequence  diagrams. 

Abstract  operation  on  a  concrete  class 

Abstract  operations  should  not  be  found  on  concrete  classes. 

Abstracted  exchange  checks 

Abstracted  exchanges  must  use  source/target  pins  with  compatible  types  to  the  conveyed  item. 

Accessibility 

The  accessibility  of  attributes,  innerclasses,  operations,  superclasses  with  be  allowable  according  to  language  guidelines. 

Activities  not  allocated  to  blocks 

An  activity  is  not  allocated  to  a  block. 
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Standards,  Style  and  Visualization 


■  As  there  are  coding  and  documentation  standards,  there  needs  to  be 
modeling  standards 

■  SysML  models,  ensure  that 

-  all  activities  have  been  allocated  to  structural  elements, 

-  item  flows  and  port  types  are  consistently  types, 

-  logical  or  abstract  elements  have  been  allocated  to  concrete  or 
physical  ones. 
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Standards,  Style  and  Visualization 


■  As  there  are  coding  and  documentation  standards,  there 
needs  to  be  modeling  standards. 

-  Quality  checks  for  model  for  code  generation 
-Classes,  operations,  attributes,  etc  are  compliant  with  the  coding 
standard 

-Complexity  metrics  can  also  be  checked  such  as  the  number  of 
attributes,  associations,  operations,  level  of  inheritance,  etc. 
(McCabe) 


01 1 1  001 1 1  00001  01  01  01  001  01 
1  00001  01  01  01  001  01 1 1  01 1  001 
1 1 1  00001  01  01  01  001  01 1 1  01 1 
01 1 1  001 1 1  00001  0 
01  01 1 1  001 1 1 
01  01  01  001 
001  0111 
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Standards,  Style  and  Visualization 


©  2010  Atego.  All  rights  reserved. 


■  Generic  quality  checks 

-  Diagram  complexity  such  as  the  “7  plus  or  minus  2”  rule,  (originally 
described  by  George  A.  Miller)  to  ensure  that  diagrams  are  readable. 

-  Ensure  that  state  diagrams  do  not  contain  dead-end  states  and 
activity  diagram  paths  can  be  executed  in  a  deterministic  fashion. 


■  As  part  of  the  adoption  of  MBE,  a  style  guide  should  be  produced  by 
the  process  owner  with  examples  to  ensure  a  consistent  approach  to 


Parametrics 


■  Used  to  express  constraints  (equations)  between  value 
properties 

-  Provides  support  to  engineering  analysis 

-e.g.  performance,  reliability,  etc 

■  Constraint  block  captures  equations 

-  Expression  language  can  be  formal 

-e.g.  MathML,  OCL  ... 

-  or  informal 

-  Computational  engine  is  defined  by  applicable  analysis 
tool 

-and  not  by  SysML 

■  Parametric  diagram  represents  the  usage  of  the 
constraints  in  an  analysis  context 

-  Binding  of  constraint  usage  to  value  properties  of  blocks 

-e.g.  vehicle  mass  bound  to  F=  m  *  a 
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Vehicle  Parametrics  BDD 


bdd  [Package]  Parametrics 


«constraint» 

Straight  Line  Vehicle  Dynamics 
constraints 

{} 

AccEq  :  Acceleration  Equation 
BrkFrceEq  :  Braking  Force  Equation 
DistEq  :  Distance  Equation  1 

VelEq  :  Velocity  Equation _ ^ 

parameters 

Bf :  Force 
m  :  Mass 
Posn  :  Position 
tf :  Friction 
tl  :  Duty  Cycle 

f 

1 


«block» 

Vehicle 

values 

Mass  : 

kg  =  2000 

Posn  : 

Position 

Power 

:  W  =  300 

«valueType» 

Loss 

«valueType» 

Mass 

«valueType» 

Time 

«valueType» 

Position 

«valueType» 

Friction 

«valueType» 

Acceleration 

«valueType» 

Duty  Cycle 

«valueType» 

Velocity 
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Z7a 


Parametrics  -  Straight  Line  Vehicle  Dynamics 


par  [block]  Vehicle  [2] 
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Measure  of  Effectiveness  /  Trade-Off  Analysis 


■  A  measure  of  effectiveness  (moe) 
is  critical  for  achieving  the  desired 

-  Cost  effectiveness 

-  Performance 

-  Communication 

-  Etc. 


represents  a  parameter  whose  value 
mission  requirements 


■  On  the  following  slide,  the  overall  cost  effectiveness  for  each  alternative 
is  defined  by  an  objective  function  that  represents  a  weighted  sum  of 
their  moe  values. 

-  For  each  moe,  there  is  a  separate  parametric  model  to  estimate  the 
value  of  operational  availability,  mission  response  time,  security 
effectiveness  and  life  cycle  cost  to  determine  an  overall  cost 
effectiveness  for  each  alternative.  It  is  assumed  that  the  moe’s  refer 
to  the  values  for  system  alternative 
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Measure  of  Effectiveness  /  Trade-Off  Analysis 


par  Effectiveness  Model  [System  Alternative  J] 
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Model  Execution 


■  UML  and  SysML  models  can  be  executed. 

-  Previously,  execution  semantics  were  not-standard 

-  Often  done  by  embedding  code  into  state  charts 

-  Acceptable  for  code  specific  models  and  programmers,  but  not  for 
systems  engineers 

■  ‘foundational  UML’  or  fUML  defines  an  execution  semantic  for  both 
activity  and  state  diagrams. 

-  fUML  specifies  a  language  independent  means  of  executing  models 

-  More  ideal  solution  for  systems  engineers  and  system  architects  who 
may  not  be  familiar  with  programming  languages. 

■  Execution  of  the  model  against  a  pre-defined  set  of  criteria  can 
determine  correct  functionality,  performance,  timing,  error  handling  and 
can  help  to  validate  use  interfaces. 

-  The  extent  to  which  they  can  do  this  effectively  determines  the  level 
of  quality  of  the  model 
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Integrating  Quality  into  the  Process 


■  Imperative  that  a  well  defined  process  be  specified  elaborating  how 
quality  checks  fit  into  the  overall  process 

-  Suggested  vs.  mandatory,  and 

-  How  updates,  modifications,  variations,  dispensations,  etc  will  be 
handled. 

■  Start  with  your  existing  process,  figure  out  where  you  would  like  to  be, 
and  determine  how  you  are  going  to  arrive  at  your  destination 
incrementally  whilst  ensuring  that  improvement  can  be  measured. 

■  Object  Oriented  Systems  Engineering  Methodology  (OOSEM). 

-  A  good  starting  point  for  defining  a  process  or  integrating  these 
concepts  into  an  existing  process 

-  Successfully  adopted  by  several  major  companies. 

-  More  information  is  available  at  the  OOSEM  website 

http://svseng.omg.org  . 
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QA  and  CMMI 


■  Process  and  Product  Quality  Assurance  (PPQA) 

-  Purpose 

-The  purpose  of  Process  and  Product  Quality  Assurance 

(PPQA)  is  to  provide  staff  and  management  with  objective 
insight  into  processes  and  associated  work  products. 

-  Specific  Practices  by  Goal 

-SG  1  Objectively  Evaluate  Processes  and  Work  Products 
-SP  1.1  Objectively  Evaluate  Processes 

-  SP  1.2  Objectively  Evaluate  Work  Products  and  Services 
-SG  2  Provide  Objective  Insight 

-SP  2.1  Communicate  and  Ensure  Resolution  of  Noncompliance 
Issues 

-  SP  2.2  Establish  Records 
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Quality  Assurance 


■  Tom  Gilb  describes  Quality  Assurance  (QA)  as  “the  generic  name  for 
any  set  of  activities,  which  have  as  their  primary  or  partial  intent,  or 
effect,  to  influence  (‘assure’)  the  quality  levels  of  a  product  or  process.” 

-  Assumes  that  modifying  the  process  will  affect  the  quality  of  the 
product  that  is  being  delivered. 

-  For  MBE  projects  it  is  essential  that  quality  criteria  for  models  be 
included  in  the  process. 

■  Jim  McCarthy  states  that  “QA’s  principal  function  is  to  continually 
assess  the  state  of  the  product  so  that  the  rest  of  the  team’s  activities 
can  be  properly  focused.” 

-  Time-consuming 

-  Potentially  error-prone 
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Automated  Quality  Assurance 


For  example,  the  following  program.  Testi.c,  contains  an  error. 

■  QA  checks  and  criteria  _ 


need  to  be  as  automated, 

1  ^include  <string.h> 

2  static  void  cpvfchar  char*  v,  unsigned  n) 

transparent,  and  painless 

3  {  int  i; 

as  possible. 

4  for  (i=0;  i<=n;  i++) 

-  No  C  programmer 

S  *v++  =  *3++; 

should  submit  his  or  her 

6  > 

code  for  review  without 

7  void  main(int  argc,  char*  argv[]) 

having  it  go  through  Lint, 

8  { 

-  No  system  designer 

9  if  (argc  !=  0) 

should  submit  a  design 

1  n  r-1  f-i <i i1  ( nl  j rn r  chrlonf 3 rnwF n "H ^ \  ■ 

for  review  without 

jlu  y v|[u j,  d  r  y  l  ,  b  Lr  ler  ii^dr  y  Y|[u  jjijr 

submitting  it  to  a  quality 

117 

check. 

Using  unt  on  Testi.c  with  the  option: 

£  lint  -errfmt=src  -Nlevel=2  Testi.c 
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Automated  Quality  Assurance 


■  To  work  effectively,  QA  needs  to  be  automated  as  much  as  possible. 

-  Manually  checking  all  of  the  tests  provided  by  Lint  would  take  a  very 
long  time,  be  error-prone,  and  suffer  from  subjectivity. 

-  Integrated  Model-Based  Quality  Assurance  requires  integrated  tools. 

These  should  provide: 

•Summary  (often  called 
dashboards)  and  detailed  views 
•Auto-correction  of  specific  types  of 
errors 

•Configurable  modelling  standards, 
visualization  of  errors,  and 
configurable  and  user-defined 
reviews  to  ensure  that  your  model 
is  complete,  consistent,  and 
correct. 
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Conclusions 


■  The  purpose  of  a  model  is  to  answer  one  or  more  questions 

-  Before  you  start,  agree  on  the  question 

■  Assessment  criteria  should  be  agreed  prior  to  starting  the  model 

■  Determine  the  right  level  of  quality  before  starting 

■  A  well-documented  process  is  essential  for  success 

-  Documentation  standards 

-  Style  standards 

-  Etc. 

■  Automation  of  assessment  checks  will  greatly  improve  productivity  and 
ensure  a  consistent  implementation 

-  Without  this  level  of  automation,  it  will  be  difficult  to  enforce  quality 
standards  across  an  organisation. 
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Questions,  Comments,  Discussion 
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